Skip to content

Implement SparseK Attention mechanism — new GGML operator with CPU backend (GPU planned next) #16817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Implement SparseK Attention mechanism — new GGML operator with CPU backend (GPU planned next) #16817

wants to merge 16 commits into from
+332 −15

Conversation

@yael-works
Copy link
Contributor

New Attention Mechanism: SparseK Attention (CPU Backend)

This PR introduces a new attention mechanism called SparseK Attention, implemented from scratch as a new operator within the GGML framework, currently with CPU backend support.

Overview

SparseK Attention is a selective and efficient attention mechanism inspired by Flash Attention, but introduces additional sparsity through:

  • Top-K filtering – keeps only the strongest attention weights.
  • Local windowing – limits attention to a configurable local context.
  • Global stride – adds periodic global connections between tokens.

Implementation Details

  • Added new operator: GGML_OP_SPARSEK_ATTN defined in ggml.h and ggml.c.
  • Implemented construction function ggml_sparsek_attn() that creates a computation node with parameters (k_top, win_local, stride_global).
  • Added full CPU backend implementation in:
    • ggml-cpu/ops.h
    • ggml-cpu/ops.cpp
    • ggml-cpu.c

The CPU version includes:

  • Scaled dot-product computation QKᵀ / √d
  • Dynamic Top-K filtering
  • Softmax normalization
  • Multiplication with V

Next Steps

Our next goal is to extend SparseK Attention to the SYCL (GPU) backend in order to:

  • Measure and compare performance between CPU and GPU implementations.
  • Optimize kernel execution for sparse attention patterns.
  • Validate correctness and scaling on Intel GPUs.

We are submitting this initial CPU implementation first to ensure review, integration, and baseline correctness before introducing GPU acceleration.

Co-Authors

Co-authored-by: Yael Shuker ([email protected])
Co-authored-by: Gitty Burstein ([email protected])

@yael-works
Add skeleton for GGML_OP_SPARSEK_ATTN (SparseK Attention): new operat…
66248d2 
…or definition and tensor creation, backend implementation pending to ggml.c/h

Co-authored-by: Yael Shuker <[email protected]>
Co-authored-by: Gitty Burstein <[email protected]>
@yael-works
Add CPU support for SparseK Attention (without performance checks)
5d6d3b7 
Co-authored-by: Yael Shuker <[email protected]>
Co-authored-by: Gitty Burstein <[email protected]>
@yael-works
Merge branch 'master' into feature/sparsek-attn-sycl
46325c7
@GittyBurstein
Copy link
Contributor

GittyBurstein commented Oct 28, 2025 •
edited
Loading

Hi @CISC and @NeoZhangJianyu,

We’d appreciate it if you could review our PR implementing the new SPARSEK Attention operator.
We ran internal validation tests we created ourselves, and all passed successfully.

This contribution was developed jointly by both of us (@yael-works and @GittyBurstein ).
Please make sure the PR reflects both contributors — if needed, we can adjust the commit authors accordingly.

Thanks in advance for your time and feedback!

@CISC
Copy link
Collaborator

CISC commented Oct 28, 2025

We are talking about this SparseK, right?

@yael-works
Copy link
Contributor Author

yael-works commented Oct 28, 2025 •
edited
Loading

yes! @CISC

@github-actions github-actions bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning labels Oct 28, 2025
@DajanaV DajanaV mentioned this pull request Oct 28, 2025
Closed
@CISC
Copy link
Collaborator

CISC commented Oct 30, 2025

You need to rebase to fix Server CI failures, also please fix whitespaces:
https://github.com/ggml-org/llama.cpp/actions/runs/18935125175/job/54060021809

Gitty Burstein and others added 4 commits October 30, 2025 13:35
CISC
CISC reviewed Oct 30, 2025
View reviewed changes
GittyBurstein and others added 3 commits October 31, 2025 01:56
@GittyBurstein @CISC
Update tests/test-backend-ops.cpp
202c5d1 
Co-authored-by: Sigbjørn Skjæret <[email protected]>
@GittyBurstein @CISC
Update tests/test-backend-ops.cpp
9712967 
Co-authored-by: Sigbjørn Skjæret <[email protected]>
@GittyBurstein @CISC
Update tests/test-backend-ops.cpp
77f4088 
Co-authored-by: Sigbjørn Skjæret <[email protected]>
@GittyBurstein
Copy link
Contributor

Hi @CISC,
Just to clarify — the failing tests are unrelated to my changes.
This PR only introduces the new SPARSEK Attention operator within GGML and doesn’t modify any existing server or inference logic.

I’d really appreciate it if you could review the code itself so we can move forward with the merge —
all SPARSEK-related tests are passing successfully.

Thanks!

@CISC
Copy link
Collaborator

CISC commented Oct 31, 2025

Hi @CISC, Just to clarify — the failing tests are unrelated to my changes. This PR only introduces the new SPARSEK Attention operator within GGML and doesn’t modify any existing server or inference logic.

Yes, as mentioned, will be resolved if you rebase, it's ok. :)

I’d really appreciate it if you could review the code itself so we can move forward with the merge — all SPARSEK-related tests are passing successfully.

So, my main challenge is where/what/when will SparseK be used? I can't recall seeing any actual implementation being used in the wild. This also means we don't really have any reference to test it against...

@GittyBurstein
Copy link
Contributor

GittyBurstein commented Oct 31, 2025 •
edited
Loading

@CISC
The current PR focuses solely on adding the SparseK Attention operator at the GGML level (CPU backend).
At this stage, it isn’t directly integrated into the model’s runtime pipeline — it’s designed as a standalone operator for experimentation and future extensions.

Once this PR is merged, the operator can be connected to higher-level use cases such as:

  • selective attention mechanisms for long-context models,

  • experimental low-latency or memory-efficient inference,

  • or research benchmarking against variants like Flash Attention or block-sparse implementations....
    Do you have any other idea that could demonstrate or validate this even better?

Thank you!!

@CISC
Copy link
Collaborator

CISC commented Oct 31, 2025

I think @ggerganov will have to weigh in on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC left review comments

@slaren slaren Awaiting requested review from slaren slaren is a code owner

@ggerganov ggerganov Awaiting requested review from ggerganov ggerganov is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yael-works @GittyBurstein @CISC